Inter-annotator Agreement for ERE annotation
نویسندگان
چکیده
This paper describes a system for interannotator agreement analysis of ERE annotation, focusing on entity mentions and how the higher-order annotations such as EVENTS are dependent on those entity mentions. The goal of this approach is to provide both (1) quantitative scores for the various levels of annotation, and (2) information about the types of annotation inconsistencies that might exist. While primarily designed for inter-annotator agreement, it can also be considered a system for evaluation of ERE annotation.
منابع مشابه
Community annotation experiment for ground truth generation for the i2b2 medication challenge
OBJECTIVE Within the context of the Third i2b2 Workshop on Natural Language Processing Challenges for Clinical Records, the authors (also referred to as 'the i2b2 medication challenge team' or 'the i2b2 team' for short) organized a community annotation experiment. DESIGN For this experiment, the authors released annotation guidelines and a small set of annotated discharge summaries. They aske...
متن کاملLearning part-of-speech taggers with inter-annotator agreement loss
In natural language processing (NLP) annotation projects, we use inter-annotator agreement measures and annotation guidelines to ensure consistent annotations. However, annotation guidelines often make linguistically debatable and even somewhat arbitrary decisions, and interannotator agreement is often less than perfect. While annotation projects usually specify how to deal with linguistically ...
متن کاملApplying Syntactic, Semantic and Discourse Constraints in Chinese Temporal Annotation
We describe a Chinese temporal annotation experiment that produced a sizable data set for the TempEval-2 evaluation campaign. We show that while we have achieved high inter-annotator agreement for simpler tasks such as identification of events and time expressions, temporal relation annotation proves to be much more challenging. We show that in order to improve the inter-annotator agreement it ...
متن کاملAnnotation manuelle de matchs de foot : Oh la la la ! l'accord inter-annotateurs ! et c'est le but ! (Manual Annotation of Football Matches : Inter-annotator Agreement ! Gooooal !) [in French]
Manual Annotation of Football Matches : Inter-annotator Agreement ! Gooooal ! We present here an annotation campaign of commentaries of football matches in French. The annotation was done from a very heterogeneous text corpus of both match minutes and video commentary transcripts. We show how the intraand inter-annotator agreement can be used efficiently during the whole campaign by proposing a...
متن کاملEnhancing the Arabic Treebank: a Collaborative Effort toward New Annotation Guidelines
The Arabic Treebank team at the Linguistic Data Consortium has significantly revised and enhanced its annotation guidelines and procedure over the past year. Improvements were made to both the morphological and syntactic annotation guidelines, and annotators were trained in the new guidelines, focusing on areas of low inter-annotator agreement. The revised guidelines are now being applied in an...
متن کامل